{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Storage\n",
    "\n",
    "**Storage** is data save/load system which provides incremental saving."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### useful functions\n",
    "\n",
    "Run this cell will load some well-used packages such as `numpy`, `pandas`, and so on.\n",
    "The running will also import some valuable functions which are written by ourselves.\n",
    "There is no magic,  see `samples/tools.ipynb` to know what will be imported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run tools.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## data saving\n",
    "\n",
    "Using `xenonpy.datatools.Storage` class to save data.\n",
    "You can save any python object and retrieve them at any time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- import necessary libraries\n",
    "\n",
    "from xenonpy.datatools import Storage\n",
    "import numpy as np\n",
    "\n",
    "# fix random seed for reproducing\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<dataset> under `/Users/liuchang/.xenonpy/userdata` includes:"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# give a dir name to the init function of 'Storage' class\n",
    "# this will init a 'Storage' object and bind to local dir `./dataset`\n",
    "test_data = Storage('dataset')\n",
    "test_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see nothing in this `Storage`, let's generate some data and save them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 0, 3, 3, 7])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generate some data\n",
    "# we will save it and load it back\n",
    "data1 = np.random.randint(0, 10, 5)\n",
    "data1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<dataset> under `/Users/liuchang/.xenonpy/userdata` includes:\n",
       "\"unnamed\": 1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save it\n",
    "test_data(data1)\n",
    "test_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'data in storage: [5 0 3 3 7]'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'original data: [5 0 3 3 7]'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<dataset> under `/Users/liuchang/.xenonpy/userdata` includes:\n",
       "\"unnamed\": 1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load it\n",
    "data_ = test_data.last()\n",
    "\n",
    "\n",
    "'data in storage: %s' % data_\n",
    "'original data: %s' % data1\n",
    "test_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data:  [5 0 3 3 7] retriever data:  [5 0 3 3 7]\n",
      "<test_data> include:\n",
      "\"unnamed\": 1 \n",
      "\n",
      "new data:  [6 8 7 9 5] retriever new data:  [6 8 7 9 5]\n",
      "<test_data> include:\n",
      "\"unnamed\": 2 \n",
      "\n",
      "retriever oldest data:  [5 0 3 3 7]\n",
      "<test_data> include:\n"
     ]
    }
   ],
   "source": [
    "# update\n",
    "data2 = np.random.randint(5, 10, 5)\n",
    "test_data(data2)\n",
    "data2 = test_data.last()\n",
    "\n",
    "print('new data: ', data2, 'retriever new data: ', data2)\n",
    "print(test_data, '\\n')\n",
    "\n",
    "# get old data\n",
    "data2 = test_data[0]\n",
    "print('retriever oldest data: ', data2)\n",
    "\n",
    "# delete all\n",
    "test_data.clean()\n",
    "print(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You can name you data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<test_data> include:\n",
      "\"data1\": 1\n",
      "\"data2\": 1 \n",
      "\n",
      "data1:  [5 0 3 3 7] ; retriever data1:  [5 0 3 3 7] \n",
      "\n",
      "data1 at index 0:  [5 0 3 3 7]\n"
     ]
    }
   ],
   "source": [
    "test_data(data1=data1, data2=data2)\n",
    "print(test_data, '\\n')\n",
    "\n",
    "# retriver by name\n",
    "data1 = test_data.last('data1')\n",
    "print('data1: ', data1, '; retriever data1: ', data1, '\\n')\n",
    "\n",
    "# by name and index or slice\n",
    "data1 = test_data['data1', 0]\n",
    "print('data1 at index 0: ', data1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combin them together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<test_data> include:\n",
      "\"unnamed\": 2\n",
      "\"data1\": 1\n",
      "\"data2\": 1\n",
      "/Users/liuchang/test/test_data-2018-06-15_21-05-07_462302.pkl.z\n"
     ]
    }
   ],
   "source": [
    "test_data(data1, data2)\n",
    "print(test_data)\n",
    "\n",
    "# dump all last to file\n",
    "test_data.dump('~/test')\n",
    "\n",
    "from pathlib import Path\n",
    "files = Path('~/test').expanduser().iterdir()\n",
    "for f in files:\n",
    "    print(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
